…feature

- Add 6 new test files covering writer-level tracking, segment creation,
  corner cases, ForwardIndexHandler reload, and integration tests for both
  offline and realtime (Kafka) ingestion paths
- Merge redundant dual-loop in TableSizeReader into a single pass over
  server info, improving performance during table size aggregation
- Fix offline integration test teardown to properly wait for table data
  manager removal before stopping servers
- Wrap second table cleanup in offline test in finally block to prevent
  resource leaks on assertion failure

…tier breakdown, and stale metadata cleanup

- Wrap flat compression fields in nested CompressionStats DTO with @JsonInclude(NON_NULL)
- Add StorageBreakdown with per-tier segment count and size (always reported)
- Add per-column ColumnCompressionDetail with aggregated sizes, ratio, and codec (MIXED when codecs differ across segments)
- Gate compressionStats on tableConfig.indexingConfig.compressionStatsEnabled; suppress from JSON when OFF
- Fix isPartialCoverage: now correctly returns true when 0 segments have stats but non-missing segments exist
- Clear stale forwardIndex.compressionCodec and forwardIndex.uncompressedSizeBytes on raw-to-dict reload
- Support null values in SegmentMetadataUtils.updateMetadataProperties to clear properties
- Add TABLE_TIERED_STORAGE_SIZE gauge; emit tier metrics always; clear compression+tier gauges when flag OFF
- Add testRawToDictClearsCompressionStats, testCompressionStatsNullWhenFlagOff, per-column/tier assertions
- Update integration tests for nested compressionStats JSON structure

…API, and comprehensive tests

- Gate uncompressed size tracking in forward index writers via compressionStatsEnabled flag
  (ForwardIndexCreatorFactory, ForwardIndexHandler, all raw index creators)
- Add per-column compression stats aggregation to server TablesResource and
  ServerSegmentMetadataReader with MIXED codec detection
- Extend TableMetadataInfo DTO with columnCompressionStats field (NON_NULL suppression)
- Fix integration test schema name mismatch for disabled-stats table
- Add 7 new test classes: IndexingConfigCompressionFlagTest, SegmentGeneratorConfigPropagationTest,
  CLPForwardIndexCreatorV2StatsTest, ServerTableSizeReaderRawBytesTest,
  TableMetadataReaderCompressionTest, TableMetadataInfoCompressionTest,
  ForwardIndexHandlerCompressionStatsTest updates

…feature flag

- Replace Map<String, ColumnCompressionStatsInfo> with List<ColumnCompressionStatsInfo> array
  containing all required fields: column, uncompressedSizeInBytes, compressedSizeInBytes,
  compressionRatio, codec, hasDictionary, indexes
- Gate columnCompressionStats in both server endpoints (TablesResource metadata,
  TableSizeResource size) on compressionStatsEnabled feature flag
- Add controller-side suppression in PinotTableRestletResource for safety against
  old servers that may still emit stats when flag is off
- Fix forward index size accumulation: use getIndexSizeFor(StandardIndexes.forward())
  directly per segment instead of cumulative variable
- Sort columnCompressionStats array by column name for deterministic output
- Update all tests and DTOs for the new array schema

…, and metadata endpoint gaps

Resolve default compression codec (LZ4/PASS_THROUGH) in BaseSegmentCreator and
ForwardIndexHandler when table config leaves chunkCompressionType null. Include
dictionary-encoded columns in columnCompressionStats with hasDictionary=true.
Clear stale controller compression metrics when no segments report stats. Suppress
zeroed compressionStats for dict-only tables. Add compressionStats summary to the
metadata endpoint aggregated from per-column data. Add tests for all fixes.

…ession

- Move columnCompressionStats to top-level field on TableSubTypeSizeDetails
  instead of nesting inside CompressionStats inner class
- Remove unused ColumnCompressionDetail inner class; use shared
  ColumnCompressionStatsInfo DTO from pinot-common
- Fix metadata endpoint field names to match size endpoint:
  rawForwardIndexSizePerReplicaInBytes, compressedForwardIndexSizePerReplicaInBytes
- Suppress compressionStats and columnCompressionStats for dict-only tables
  and when feature flag is OFF
- Dictionary columns report -1 for uncompressedSizeInBytes to distinguish
  from zero-size raw columns on both size and metadata endpoints
- Use LinkedHashSet for index deduplication in per-column aggregation

…adata endpoint

- Create CompressionStatsSummary DTO in pinot-common with
  rawForwardIndexSizePerReplicaInBytes, compressedForwardIndexSizePerReplicaInBytes,
  and compressionRatio
- Create StorageBreakdownInfo DTO in pinot-common with per-tier count and size
- Add both as @JsonInclude(NON_NULL) fields on TableMetadataInfo with
  backward-compatible constructors
- Server computes compressionStats summary and storageBreakdown during segment
  iteration and includes them in TableMetadataInfo response
- Controller aggregates both fields across servers in ServerSegmentMetadataReader
  (divides by numReplica like other fields)
- Remove manual addCompressionStatsSummary() JSON manipulation from
  PinotTableRestletResource; controller now just strips fields when flag is OFF
- Fix use-after-release: tier accumulation moved inside try block before
  segments are released in the finally block

…racking accuracy

- Remove ObjectNode.remove() pattern from PinotTableRestletResource; pass
  compressionStatsEnabled flag through TableMetadataReader and
  ServerSegmentMetadataReader so DTOs are constructed with null at creation
  time when the flag is OFF (storageBreakdown always preserved)
- Add segmentsWithStats, totalSegments, isPartialCoverage to
  CompressionStatsSummary so metadata endpoint has identical JSON schema
  to the size endpoint; populate during server and controller aggregation
- Fix VarByteChunkForwardIndexWriterV4 uncompressed size tracking: track
  raw value byte lengths in putBytes() instead of buffer.remaining() in
  write() which included chunk-format header overhead
- Move segment stats counting inside the try block in TablesResource to
  avoid accessing segment metadata after segments are released
- Add test for compression stats suppression when flag is disabled

Old segments lacking uncompressed size metadata (pre-flag segments) were
contributing their compressed forward index size to the denominator while
adding nothing to the numerator, deflating the compression ratio. Now
only dictionary-encoded columns enter the compressed-only accumulation
path; raw columns on old segments without codec/uncompressed data are
skipped entirely from both numerator and denominator.

- Track uncompressed size per value in putInt/putLong/putFloat/putDouble
  (FixedByteChunkForwardIndexWriter) and putBytes (VarByteChunkForwardIndexWriter)
  instead of using chunk buffer remaining bytes which overcounts due to
  chunk-internal offset tables
- Gate server size endpoint compression field collection on feature flag
  so no metadata access or accumulation occurs when flag is OFF
- Exclude old raw segments lacking a persisted compression codec from
  per-column stats to prevent sentinel values leaking into aggregation
- Guard Math.max in per-column accumulation against INDEX_NOT_FOUND (-1)
  sentinel values
- Preserve per-column compression stats for dict-only tables even when
  no segments have raw forward index data (segmentsWithStats == 0)
- Rename TABLE_COMPRESSION_RATIO_HUNDREDTHS gauge to
  TABLE_COMPRESSION_RATIO_PERCENT for consistency
- Hoist IndexService.getInstance() outside per-column inner loop

- Fix negative compression ratio for dict columns in metadata endpoint:
  require both uncompressed > 0 and compressed > 0 before computing
  ratio in ServerSegmentMetadataReader (per-column and summary level)
- Track emitted tier gauge keys per table in TableSizeReader so stale
  tier-suffixed gauges (tableName.tierKey) are removed when tiers
  disappear or table is deleted via SegmentStatusChecker cleanup
- Resolve CLP V2 actual compression type (ZSTANDARD) from the
  CompressionCodec before falling back to ForwardIndexConfig's
  chunkCompressionType, which maps all CLP variants to PASS_THROUGH

Extract resolveCompressionType() as a shared utility in ForwardIndexType
that correctly maps CLP codec variants to their actual compression types
(CLPV2/CLPV2_ZSTD → ZSTANDARD, CLPV2_LZ4 → LZ4, CLP → PASS_THROUGH).
This fixes ForwardIndexHandler using incorrect compression types during
codec changes and dict-to-raw conversions. BaseSegmentCreator now uses the
same shared method, handling nullable fieldType for schema evolution cases.

Also document CLPForwardIndexCreatorV2.getUncompressedSize() semantics:
returns pre-compression sub-stream byte total, not original UTF-8 length.

…umns

Size endpoint flag-OFF path now uses the 5-arg SegmentSizeInfo constructor
to pass through tier information, fixing storageBreakdown flattening all
segments into the "default" tier when compression stats are disabled.

Metadata endpoint aggregation now skips columns from old raw segments that
have no persisted compression codec and no dictionary, preventing zero-filled
entries from appearing in per-column compression stats.

Dictionary columns report -1 as uncompressed forward index size. When
aggregating across replicas, skip accumulation for negative sentinels
(using >= 0 guard) and reconstruct -1 in the output for dict columns
that have no real uncompressed data.

Also guard compressionStatsSummary construction on segmentsWithStats > 0
after replica division, avoiding a degenerate summary when integer
division rounds the count to zero.

…stats

Tier gauge emission was passing the tier-suffixed key (e.g.
"myTable_OFFLINE.coldTier") to isLeaderForTable(), which expects the
base table name. Hoist the leader check to use the canonical
tableNameWithType before the tier loop, then emit gauges directly.

On the server metadata endpoint, move computeIfAbsent for per-column
compression accumulators inside the conditional branches so old raw
segments without a persisted codec no longer create zero-filled entries.

- Reorder FieldConfig import in BaseSegmentCreator to satisfy spotless
- Replace ColumnMetadata.INDEX_NOT_FOUND with UNAVAILABLE (upstream rename)
- Replace deprecated RandomStringUtils.randomAlphanumeric with secure().nextAlphanumeric

…Summary

- ColumnMetadataImplTest: add two tests verifying the new compression
  stats fields (FORWARD_INDEX_UNCOMPRESSED_SIZE, FORWARD_INDEX_COMPRESSION_CODEC)
  round-trip through fromPropertiesConfiguration, and that old segments
  without those keys return UNAVAILABLE/-1 and null respectively.
- CompressionStatsSummaryTest: new test class covering all getters,
  full/partial coverage flag, JSON round-trip, and unknown-field
  backward compatibility for the new DTO.

- ForwardIndexTypeTest: add testResolveCompressionType covering all 6 branches
  (CLP→PASS_THROUGH, CLPV2/CLPV2_ZSTD→ZSTANDARD, CLPV2_LZ4→LZ4, regular codec,
  field-type fallback, null fieldType)
- SegmentMetadataUtilsTest: new test verifying null-value map entries clear the
  property vs non-null entries overwriting it
- ColumnCompressionStatsInfoTest: new test covering all getters, JSON round-trip,
  null codec/indexes, hasDictionary, and unknown-field tolerance
- StorageBreakdownInfoTest: new test covering TierInfo getters, multi-tier
  round-trip, empty map, and unknown-field tolerance on both types
- TablesResourceTest: verify columnCompressionStats and compressionStats are null
  when compressionStatsEnabled is false
- TableSizeResourceTest: verify columnCompressionStats is null per-segment when
  compressionStatsEnabled is false

…ssertion style

…rageBreakdown aggregation and dict sentinel path

…he#18470

…ssedSize coverage

…t SPI

The method was abstract, which would break any external IndexCreationContext
implementor on upgrade. Provide a default returning false (feature-off) to
preserve backward compatibility for plugins and external implementations.

Also document the mutable CompressionStats/StorageBreakdown inner classes as
intentional accumulators separate from the immutable pinot-common DTOs.

…oveTableGauge(name, key, gauge)

Tier storage gauges were emitted as setValueOfTableGauge(tableName + "." + tierKey, gauge, value),
concatenating the tier key into the tableName slot. This produces a non-canonical gauge name that
is invisible to any code querying gauges by table name. Use the correct 3-argument overloads
setOrUpdateTableGauge(tableName, tierKey, gauge, value) and removeTableGauge(tableName, tierKey, gauge)
throughout emitTierMetrics and clearTierMetrics.

The _trackUncompressedSize default-flip (C4.13) is deferred: writers are used directly in tests and
tooling without going through ForwardIndexCreatorFactory, so changing the default breaks those callers.
The existing factory-based path already sets the flag correctly via setTrackUncompressedSize(context.isCompressionStatsEnabled()).